Home device application programming interface

ABSTRACT

The present disclosure contemplates a variety of improved methods and systems for enabling developers to program and control a variety of disparate devices coupled to the framework and/or ambient operating system. The Home API can expose operation of the devices in a natural and intuitive manner, facilitating control of unknown devices by high-level commands.

CLAIM FOR PRIORITY

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/449,492, entitled “HOME API,” by Roman et al., and filed on Jan.23, 2017, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure contemplates a variety of improved methods andsystems for enabling developers to program and control a variety ofdisparate home devices coupled to the framework and/or ambient operatingsystem.

BACKGROUND

The Internet of Things (IoT) allows for the internetworking of devicesto exchange data among themselves to enable sophisticated functionality.For example, assistant devices configured for home automation canexchange data with other devices to allow for the control and automationof lighting, air conditioning systems, security, etc. Existing solutionsrequire users to select individual devices and ascribe settings to themone-by-one, potentially within a menu format.

SUMMARY

The present disclosure contemplates a variety of improved methods andsystems for enabling developers to program and control a variety ofdisparate home devices coupled to the framework and/or ambient operatingsystem (AmbientOS). This home application programming interface (API)exposes the operation of the home devices in a natural and intuitivemanner, facilitating control of unknown devices by high level commands.The home devices are contextually responsive to API commands.

Some of the subject matter described herein includes a method forseamlessly and automatically building an application using anapplication programming interface (API) to interface with one or moredevices within an environment with an assistant device having artificialintelligence (AI), comprising: receiving, via a processor, a graphicrepresenting a layout of one or more graphical user interface (GUI)elements for the application to instruct the assistant device to controlthe one or more devices within the environment; identifying a section ofthe graphic corresponding to the one or more of the GUI elements that isto be a selectable item on the application to implement a functionalityto be performed within the environment using the assistant device, thesection identified based on a change in color between a background colorof the graphic and a section color of the graphic; identifying graphicalcontent within the section; determining object characteristics of thegraphical content within the section corresponding to the one or more ofthe GUI elements that is to be the selectable item on the application toimplement the functionality to be performed within the environment usingthe assistant device, the object characteristics including a type ofobject corresponding to the graphical content, a position of thegraphical content within the section, a size of the graphical content inrelation a size of the section, a relationship between the position ofthe graphical content with other graphical content within the section,or a graphical content color; identifying textual content within thesection; determining text characteristics of the textual content, thetext characteristics of the textual content including one or more of ameaning of the textual content, a size of the textual content, a font ofthe textual content, a relationship between a position of the textualcontent and the position of the graphical content, or the position ofthe textual content within the section; determining the functionalityassociated with the section based on the text characteristics of thetextual content and the object characteristics of the graphical content;identifying the one or more devices capable of performing thefunctionality within the environment; and generating the applicationhaving the section of the graphic, wherein the section when selectedinstructs the assistant device to cause the identified one or moredevices to perform the functionality.

Some of the subject matter described herein includes a methodcomprising: receiving, via a processor, a graphic representing a layoutof one or more graphical user interface (GUI) elements for anapplication to instruct an assistant device to control one or moredevices within an environment; identifying a section of the graphiccorresponding to the one or more of the GUI elements that is to be aselectable item on the application to implement a functionality to beperformed within the environment using the assistant device; determiningcharacteristics of a graphical content of the graphic wherein thegraphical content can include a depiction of an object or a text;determining the functionality and the one or more devices associatedwith the section, the functionality to be performed within theenvironment using the assistant device based on the characteristics ofthe graphical content; and generating the application having the sectionof the graphic, wherein the section when selected instructs theassistant device to cause the one or more devices to perform thefunctionality.

Some of the subject matter described herein includes an electronicdevice, comprising: one or more processors; a database storingcharacteristics of graphical content, a functionality, and devicecategories; and memory storing instructions, execution of which by theone or more processors cause the electronic device to: receive a graphicrepresenting a layout of one or more graphical user interface (GUI)elements for an application to instruct an assistant device to controlone or more devices within an environment; identify a section of thegraphic corresponding to the one or more of the GUI elements that is tobe a selectable item on the application to implement the functionalityto be performed within the environment using the assistant device;determine characteristics of the graphical content of the graphicwherein the graphical content can include a depiction of an object or atext; determine using the database, the functionality, and the one ormore devices associated with the section, the functionality to beperformed within the environment using the assistant device based on thecharacteristics of the graphical content and the device categories; andgenerate the application having the section of the graphic, wherein thesection when selected instructs the assistant device to cause the one ormore devices to perform the functionality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of the assistant device connected to adevice;

FIG. 2 illustrates an embodiment of determining device categories;

FIG. 3 illustrates an embodiment of a Home API graphic user interface(GUI);

FIG. 4 illustrates an embodiment of a Personal Home API GUI;

FIG. 5 illustrates an embodiment of an application development screen;

FIG. 6 illustrates an example of an assistant device; and

FIGS. 7a-7l illustrate embodiments of an assistant device.

DETAILED DESCRIPTION

This disclosure contemplates a variety of improved methods and systemsof a development environment associated with an assistant device. In atleast one embodiment, the development environment is the home deviceapplication programming interface (Home API). The described developmentenvironment can include procedures and tools for developing, debugging,and testing applications, apps, and/or commands to customize a user'shome. For example, a developer wanting to create an application forcontrolling all lighting devices in the home can use the developerenvironment to create the app. In at least one embodiment, the home canbe the environment surrounding the assistant device such as a house, anoffice, a car, and/or a store. The generated applications can includemobile applications (apps).

A home can include many different electronic devices capable ofproviding different functionalities. For example, a home can includelights, thermostat, a smart TV, and a coffee maker. These devices can begiven different functionalities such as turning on the lights, changingthe temperature, turning off the news, and making coffee. The devicescan be configured to connect (e.g., via Wi-Fi, Bluetooth, etc.) to theassistant device, and the assistant device can be configured to controlthe devices. For example, a user can tell the assistant device “pleaseturn off the TV”; in response, the assistant device can send a requestto the TV, causing the TV to be turned off. In another example, the usercan tell the assistant device “please turn off all the lights in thehome”; in response, the assistant device can send a request to all thelights, causing all the lights in the home environment to be turned off.This configuration can save the user time by controlling devices throughthe assistant device instead of physically operating the individualdevices.

The developer environment allows users and/or developers to build on theassistant device infrastructure by providing development tools,routines, protocols, and resources to simplify developing, debugging,and testing applications, apps, and/or commands to customize theirhomes. For example, the developer can use the home API to create anapplication for a smartphone (communicatively coupled to the assistantdevice) which can control all the lights in the home.

FIG. 1 illustrates an embodiment of the assistant device connected toother devices. The assistant device 101 is connected to one or moredevices. Devices can include one or more connected devices such asgarage doors 102, speakers 103, lights 104, thermostats 105, and mobiledevices 106. Devices can also include household appliances such asfreezers, ovens, air purifiers, washers, and dryers. In at least oneembodiment, the devices are devices such as wearable devices, and/orrobots such as robotic vacuums.

Devices can be connected to the assistant device physically and/or via awireless signal. In at least one embodiment, the wireless signal caninclude one or more of Long-Term Evolution (LTE), LTE-Advanced, Wi-Fi,Bluetooth, ZigBee, EnOcean, Personal area networks, TransferJet,Ultra-wideband, WiMAX, HiperMAN, Li-Fi and/or infrared radiation (IR).

Each connected device can be controlled using an associated adapterand/or driver. The adapter can be device-specific and, in at least oneembodiment, functionality specific. For example, a thermostat may havean associated adapter which the assistant device can use to operate orcontrol the thermostat. In an example, the assistant device may have anadapter for each of the functionalities of the thermostat (e.g., raisetemperature, lower temperature, turn on fan, etc.). The adapters can bestored on local resources (e.g., assistant device memory, assistantdevice external memory, etc.) and/or on remote resources (e.g., cloud,etc.).

In at least one embodiment, the assistant device can access a databasewhich stores a list of connected devices and one or more of theassociated adapters, one or more voice activatable commands, activitytypes, device descriptions, device categories, and/or device location.The database can also be stored on the local resources of the assistantdevice. In at least one embodiment, the database can be storedexternally from the assistant device. For example, the database entryassociated with the thermostat can be a reference to the thermostatadapter, voice activatable commands (e.g., “set temperature,” “turn onfan,” etc.), device description (e.g., “thermostat”), device category(e.g., “temperature regulation”), and/or device location (e.g., “livingroom,” etc.).

Connected devices can be controlled by the user via the assistantdevice. For example, a user can instruct the assistant device verbally“assistant device, please turn on the television.” In response, theassistant device can parse the user instruction using a speechrecognition algorithm. The voice activatable command “turn on” and thedevice description “television” can be used to identify the specificdevice. The assistant device can then transmit a command to thetelevision requesting it to perform the function of “turning on,”thereby causing the television to turn on.

In at least one embodiment, the speech recognition algorithm can includeone or more of Hidden Markov models, dynamic time warping-based speechrecognition, neural networks, deep feedforward neural networks,end-to-end Automatic Speech Recognition (ASR), and/or other similaralgorithms.

In some implementations, a home assistant device with artificialintelligence (AI) capabilities can receive audio data as voice inputfrom a user and provide audio data as voice output, thereby respondingto the user via spoken language. The home assistant device can alsomanipulate other devices within the home, for example, turn on or offdevices (e.g., light bulbs), increase volume of a television, adjust thetemperature setting of a thermostat, etc.

For example, the home assistant device can be requested to play backmusic using its own speaker or speakers elsewhere within the home, or acombination of both. In some implementations, as the home assistantdevice or other devices with speakers are playing back music, thecharacteristics of that music can be adjusted based on the activities ofpeople within the environment. For example, if two people are having aconversation, then this can be detected by the home assistant device(e.g., via its microphones, or the microphones of other devices withinthe home that the home assistant device has access to) and the volume ofthe music can be adjusted such that it is lower, allowing the people toeasily have a conversation with background music being unobtrusive. Insome implementations, the speed of the playback of the music can beadjusted (e.g., made slower), play back of some frequency ranges of themusic can be changed (e.g., sounds within a lower bass frequency rangemight be reduced in volume while sounds within a higher treble frequencyrange might remain the same as before the conversation was started),etc.

In another example, the home assistant device can determine when a useris getting ready in the morning to leave the home, for example, to go towork or a social engagement. For example, the home assistant device candetermine whether a user has awaken by picking up sounds of the userrustling out of bed, brushing his or her teeth, etc. In another example,if the bathroom light is turned on within a threshold time range (e.g.,6:00 AM to 10:00 AM), then this determination by the home assistantdevice can indicate that a user has awakened. The home assistant devicecan then access a user's calendar, either online in the cloud or storedlocally on the home assistant device or the user's smartphone, anddetermine when the user's first appointment for the day upon waking upis. The home assistant device can determine the location of the home andthe location of where the user needs to go (e.g., the workplace on aweekday morning) and then determine the traffic between the twolocations and determine what time the user should be leaving in order toarrive on time. When the user is still within the home and approachingthe time to leave, the home assistant device can inform the user in avariety of different ways. For example, it can play a noise, begin toblink its screen or light emitting diodes (LEDs) or other lighting, oreven cause other devices to function differently, for example, bycausing lights in the room that the user is determined to be within tostart blinking. For example, if the user is within the restroom, thenthe restroom lights can begin blinking, dim, or cycle betweendim-to-normal brightness, the other appliances in the restroom can beoperated (e.g., the toilet may flush, the toothbrush can begin vibratingby actuating accelerometers), etc. as instructed by the home assistantdevice. In some embodiments the assistant device can include an alarmfeature which can control devices in the environment. For example, theassistant device can cause the lighting devices in the home to blinkwhen the alarm is triggered. In some embodiment the assistant device canidentify the location of the user associated with the alarm and onlycause the lights in the user's vicinity to blink. In some embodimentsthe assistant device can account for the time it takes a user to getready to leave home. For example if user A takes 30 minutes to prepareto leave home, the assistant device can notify the user A 30 minutesprior to required departure.

In some implementations, the home assistant device can determine thatthe user has deviated from an expected schedule of behaviors thattypically occur within the home. For example, if the user usuallyoperates a toothbrush within ten minutes of waking up, this can bedetermined. If the user later wakes up, enters the restroom, leaves therestroom, and then begins to make coffee, the home assistant device candetermine that the user did not operate the toothbrush. The homeassistant device can then alert the user, for example, by generating anotification on its display screen via a graphical user interface (GUI),send the user a text message receivable via their smartphone, or operatedevices to alert the user similar to the features described aboveregarding informing the user when to leave to arrive to a location ontime. The assistant device can also identify significant dates in auser's life and can create associated tasks for the user. For example ifa user has a wedding anniversary on September 30th, the assistant devicecan add a task of “buy anniversary gift” prior to September 30th.

In some embodiments the assistant device can be initiated by a tap, orglance. The tap can be identified by the assistant device via textilefeedback on the assistant device screen. In some embodiments the tap canbe identified by an accelerometer in the assistant device detecting aphysical movement. In some embodiments the tap can be identified byresistive touch screen technology, capacitive screen technology, orfrustrated total internal reflection technology.

FIG. 2 illustrates an embodiment of determining device categories. Thedevices can collect and/or generate a lot of data. The device data caninclude and/or be used to derive contextual data. Device data caninclude user commands to the assistant device such as “please dim livingroom lights when a television is turned on.” In an example, theassistant device can store the request by the user to have thetelevision turned on. The user requests and interactions with devicescan be stored as device data information. In at least one embodiment,the assistant devices store only information collected from the devicessuch as a request to have the lights turned on. In at least oneembodiment, device data can include one or more of sensor data, usermanual command, user verbal command, user gesture command, settingsdata, status data, environmental data, and/or other data collected bythe device and/or assistant device.

In at least one embodiment, the device data information is collected asdemonstrated at step 201, and category information is derived using amachine learning algorithm 202. Categories can be based on derivedfacets, features, contextual information, relationship informationand/or semantic information. The machine learning algorithm can includeone or more of decision tree learning, association rule learning,artificial neural networks, deep learning, inductive logic programming,support vector machines, clustering, Bayesian networks, reinforcementlearning, representation learning, similarity and metric learning,sparse dictionary learning, genetic algorithms, rule-based machinelearning, learning classifier systems, supervised learning, unsupervisedlearning, semi-supervised learning, clustering algorithm, and/orclassification algorithm.

For example, collected device data of a thermostat can include thetemperature change requests and timestamps, and collected device datafrom the television can include timestamps of when the device was used.The machine learning algorithm can analyze the collected device data todetermine patterns. Patterns can include correlations between collecteddevice data; for example, it can be determined that the temperature onthe thermostat is always raised when the television is on. In anembodiment, the device data information can be used to determine thatspecific devices are in the same room or in different rooms. In anexample, if a light device is always turned on when the refrigeratordevice is opened, it can be determined that the two devices are related.In another example, the device data information can be used to determinewhich devices are light devices.

The received data information can further be used to generate and/orassign categories. In at least one embodiment, the devices arecategorized using device data information. The devices and/or thecorresponding facets, features, contextual information, relationshipinformation, and/or semantic information can be used to determinecategories of devices 203. For example, all devices which control lightscan be categorized as “lighting devices,” and all devices which aredetermined to be in the master bedroom can be grouped as “master bedroomdevices.” In at least one embodiment, the devices can be members ofmultiple categories. The categories of devices and/or the correspondingfacets, features, contextual information, relationship information,and/or semantic information can be performed via a machine learningalgorithm. The categories can include device location, activity type,and device categories.

For example, based on the use and context, it can be determined thatboth the thermostat and the television are located in the living room.It can also be determined that the thermostat and television are usedall day on weekends but only in the evenings during the week. Thisinformation can be saved as a weekend or weekday evening activity type.In some embodiments, the assistant device can determine that the lightsin the kitchen and the lights in the living room should be assigned tothe illumination device category. In at least one embodiment, a user canmanually enter, delete, and/or edit the categories and names of devices.

In at least one embodiment, the functionality information is collectedfor each device and is made available via the Home API. In at least oneembodiment, the Home API includes a list of categories including devicetypes and/or locations. Categories can include one or more of lights,home appliances, audio devices, wearable devices, user mobile devices,plant watering devices, televisions, and/or locks. Categories caninclude locations such as home, office, bedroom, master bedroom, livingroom, dining room, kitchen, and/or basement. Additionally, the devicetypes and categories can include the respective functionalities such as“open” and “closed” functionalities of blinds, which can be associatedwith a window coverings category. When a device is connected to one ormore assistant devices, it is associated with one or more categories.Therefore. a developer does not need to know the specific devices in theone or more homes but instead can indicate the rules and commands. Forexample, a developer can create a rule that when a certain button on anapplication is pressed, all devices that intersect the categories of thebedroom and light be switched to the functionality “off.” The category,device, and/or functionality information can be made available todevelopers on the Home API. Therefore, the developer does not need toknow the devices in the home but instead can rely on the categorieswhich can, in the future, have devices assigned. For example, thedeveloper does not need to know the exact devices which the user willinstall in the bedroom but instead can rely on the knowledge thatdevices will be connected and assigned to the bedroom category.

In at least one embodiment, the device data collected at 201 iscollected from all devices that can potentially be connected to the homeassistant. For example, when a new device is introduced to the market(e.g., a new device is launched by a company, etc.) the device data iscollected about the device, and one or more categories are assigned tothe device. The device data can include data released by the device bythe manufacturer and/or another party. The devices and the associatedcategories can be stored in a database. In at least one embodiment, thedatabase is publically available. In at least one embodiment, thedatabase is available to all developers of a Home API. In at least oneembodiment, the devices are categorized for the environment such as awork environment, home environment, and car environment.

FIG. 3 demonstrates an embodiment of a Home API graphic user interface(GUI). A developer can use the GUI 301 to create an app. The GUI canallow a developer to generate one or more graphics or import one or moregraphics. The graphic can include a layout of graphical user interfaceelements for the application to instruct the assistant device to controlone or more devices within the environment. The graphic can be generatedby creating a background and/or elements.

The developer does not need to know the devices in a specific homeenvironment and can select the devices' categories for identification.For example, a developer intending to create an application which turnsall the lights off in the kitchen does not need to know the list of alldevices that can be connected as lights in the kitchen, but instead canselect the intersection 303 of categories 302, light devices and kitchendevices. The Home API GUI can also allow the developer to select thefunction 304 of the selected device categories. For example, thedeveloper can select the function to turn off the devices that meet thecriteria of intersection of light devices and kitchen devices.

The developer can configure applications without knowledge of thedevices in the home environment. The developer can use categories toidentify devices and the associated functions. Categories can includelocation categories such as kitchen, bedroom, master bedroom, bathroom,garage, child's bedroom, guest bedroom, dining room, etc. Categories canalso include device categories such as lighting devices, cookingdevices, entertainment devices, television devices, office devices, etc.The devices in the home environment that meet the criteria identified inthe application by the developer can be controlled using the app. Forexample, although a developer of the application does not know whichlight device is in the user's kitchen, the developer can select kitchenand lighting device categories during the application development, andthis will allow the application to control the functionalities of thekitchen light which is associated with the kitchen and light categoriesin the home environment. In at least one embodiment, the setup processof a device in the home includes identifying one or more devicecategories associated with the device. These categories can then be usedto identify the devices.

Furthermore, functionalities 304 can be assigned to devices.Functionalities can include any functionality a device is capable ofperforming such as turn on, dim, turn off, make coffee, open, etc. Thefunctionalities displayed on the Home API 301 screen can be dynamicallyadjusted depending on the device category selected. For example, when acoffeemaker device category is selected, the functionalities options canbe adjusted to show functionalities of coffeemaker devices such as makecoffee, grind coffee, turn off, keep warm, etc.

In at least one embodiment, the Home API allows the developer to createrules with two or more IoT devices. For example, the developer cancreate a rule having a condition of a specific wearable IoT devicereporting over 10,000 daily steps, and that when a user device is withinproximity of 10 feet from the home device, the condition will triggerthe assistant device to play the song “Gonna Fly Now” via an IoT devicespeaker.

The rules can include services such as timer services, video monitoringservices, audio monitoring services, signal monitoring services, newsmonitoring, conflict monitoring services, or a combination thereof. Aselectable item can be configured to delay (using a timer service), whenselected, the performance of the functionalities by the one or moredevices until a specific time is identified on a timer. For example, adeveloper can program a selectable item called “start making coffee in10 minutes” to cause the coffee maker to make coffee 10 minutes afterthe selectable item is selected by the user. The timer service also canallow the developer to create a selectable item, which when selected,causes the assistant device to cause a device to perform a functionalityfor a set time (e.g., 30 seconds, 5 minutes, 2 hours, etc.), and thencause the assistant device to send a request to the device to stopperforming the functionality. For example, a developer can create aselectable item which when selected turns the lights on for only 15minutes, and after the 15 minutes elapse, causes the lights to turn off.

The rule can also include a video monitoring service. A selectable itemcan be configured, when selected, to cause devices to perform afunctionality until a specific video monitoring condition is met. Forexample, a developer can program the selectable item, when selected, tocause an air-conditioner to turn off until someone comes home. Selectingthe selectable item having a rule associated with video monitoring cancause the video monitoring service to monitor the environment for thespecified condition (e.g., a person coming into the environment, etc.),and then, when the condition is met, to perform a functionality. In someembodiments, the video monitoring and the timer services can be combinedin a rule condition. For example, the selectable item can be programmedto monitor the video input for a specific period of time (e.g., 45minutes, etc.).

The signal monitoring service can monitor for devices in the home (e.g.,mobile device entering the home), changes of status of devices (e.g.,television turned on, thermostat turned off, etc.), or status of devices(e.g., thermostat temperature reached 72 degrees, etc.). The newsmonitoring service can monitor the news such as the weather channel,current events, etc. The news monitoring service can monitor one or morenews services for specific situations such as natural disasters (e.g.,hurricanes, tornadoes, earthquakes, etc.), police activity (e.g.,lockdown of a neighborhood), public service announcements (e.g.,brownout warnings, etc.), and/or news events (e.g., brownouts, etc.).For example, a developer can configure a selectable item that can, whenselected, turn on all the lights in the house unless a brownout warningis detected by the news monitoring service, in which case thefunctionalities associated with the selectable item are not beperformed. In some embodiments, the developer can configure the warningto be displayed to user, and the user can select to ignore it andperform the functionalities associated with the selectable item.

The conflict monitoring service associated with a rule can monitor thestatus of all devices in the home and determine whether the rule beinginvoked causes a conflict with functionalities being performed in theenvironment, an operation in the home, and/or the settings of one ormore devices. For example, a developer can create a rule associated witha selectable item which does not allow for the bathroom lights to beturned off when the shower is running. Thus when a selectable item “turnoff all lights” is selected, the conflict monitoring service determinethat the shower is on. In the instant example, the developer canconfigure the rule to perform the functionalities associated with somedevices (e.g., turn off all lights except bathroom light) or canconfigure the rule to not implement any functionalities associated withthe selectable item when a conflict is detected. In some embodiments,the developer can configure the rule to display the conflict to the userwhen the conflict is detected. In some embodiments, these services areimplemented using microservices.

In at least one embodiment, each selected rule and/or functionality hasa corresponding mechanism which allows the assistant device to providefunctionalities such as providing instructions for connected devices.Mechanisms can include remote invocation, service registration,publication, and/or subscriptions. The mechanisms can have correspondingpolicies. The policies can be responsible for deciding how to addressevents such as network errors and services behaving erratically. In atleast one embodiment, the Home API allows the developer to configure themechanisms and/or policies.

Furthermore, the rules, functionalities, and/or services can implementedusing microservices. Microservices can be addressable remote objectsthat encapsulate a state and expose a set of functionalities with theirassociated parameters that can optionally return parameters uponinvocation. The microservices can communicate by fully asynchronousmessage passing. Microservices can be reactive or proactive. A reactivemicroservice waits for remote invocations to execute an action, andoptionally returns some values. For example, a microservice associatedwith a rule requiring the lights to be turned on when an icon is pressedcan be a reactive microservice.

A proactive microservice can handle remote invocations, but it can alsoexecute functionality in the background (e.g., timer service, voicerecognition, etc.). For example, a reactive microservice can beassociated with a rule requiring the coffee maker device to keep thecoffee warm for 15 minutes (e.g., timer service counts 15 minutes toturn off the coffee device, etc.). A voice recognition service can be abackground service that proactively listens to the microphone andanalyzes the audio. Microservices can be packaged as componentsassociated with device adapters. In some embodiments, the system can beconfigured to perform the initiated functionality or rule when amicroservice is not responsive. In some embodiments, the microservice isstateless. In an embodiment, the application can be configured not tofail when one or more microservices are unavailable.

FIG. 4 demonstrates an embodiment of the Personal Home API GUI. A HomeAPI can be available to individuals looking to build applications tocustomize their homes. In at least one embodiment, an owner of anassistant device can build an application using the Personal Home APIGUI 401 in her home environment. The developer (e.g., owner) can use thePersonal Home API to view available devices 403 such as kitchen ceilinglight and kitchen table lamp. The developer can generate applicationsand customize the home environment. In at least one embodiment, thedeveloper can narrow down the list of available devices by selecting acategory 402.

A developer can select a view type such as the device categories 402(e.g., device location, device function type, etc.). In the demonstratedexample, the lighting device and the kitchen device categories areselected, and all the devices that meet the criteria are displayed; seeelement 403. Once the element is displayed, the developer can select thedevice. In at least one embodiment, after selecting a device, thedeveloper is given an option to select the available set of functions ofthe device. For example, after selecting kitchen table lamp, thedeveloper can be prompted to select the function of the kitchen tablelamp such as “lights on.” The developer can select the function and/ordevice and associate it with a button and/or command on a screen.

In at least one embodiment, the Home API can allow the user to set uprules and/or conditions. The developer can select and/or enterconditional statements such as if, then, when, where, and equal. In atleast one embodiment, the developer can configure rules using Booleanoperators such as and, or, and not. For example, the developer cancreate a rule that when specific conditions are met (e.g., all lightsare on, etc.) a device should perform a function (e.g., smart curtainsclosed, etc.). The rules can be created by a developer selectingdevices, functions, operators and/or statements from a menu on agraphical user interface. The rules can be set to automatically performa specific task when a condition is met. In at least one embodiment, therules can be created by the developer writing code.

In at least one embodiment, the Home API allows the developer to createrules with two or more IoT devices. For example, the developer cancreate a rule having a condition of a specific wearable device reportingover 10,000 daily steps, and when a user device is within a proximity of10 feet from the home device, the condition will trigger the assistantdevice to play the song “Gonna Fly Now” via a device speaker. In someembodiments, the rules are automatically generated based on the selectedfunctionality and/or device category.

Rules can be automatically generated using a rule definition databasehaving information about device, device locations, device types,functionalities, and associated attributes. The attribute information inthe rules definition database can include attribute information aboutwhich devices, device locations, device types, and functionalitiesconflict. Attribute information can indicate whether the devices and/orlocations are security related (e.g., locks, etc.), potentiallydangerous (e.g., can cause fire, etc.), incompatible (e.g., devices orfunctionalities that should not be used at the same, etc.), and/or timesensitive (e.g., used only between certain hours, used only for a setperiod of time, etc.).

The attribute information can determine the selectable item rulesgenerated. An attribute associated with thermostat devices can specifythat all thermostat devices in the environment should be set to similartemperature. For example a conflicting functionality can be setting ahigh temperature on one thermostat and setting the air-conditioner on asecond thermostat in the same room. Other attribute information canexist about devices and functionalities, for example, the devicesassociated with security can require authentication. In someembodiments, only the “unlock” functionality of a security device canrequire authentication. For example, when a developer generates aselectable item “unlock lock” with the device type “lock” andfunctionality “unlock,” a rule can automatically be generated requiringauthentication. The rule can require that the video monitoring serviceis initiated in response to the “unlock lock” selectable item beingselected, and the video monitoring service can determine whether theindividual selecting the item is a primary user and/or member of theprimary user group associated with the home. In some embodiments thevideo monitoring service can be configured to monitor the use of the“unlock lock” selectable item and report any unusual activity to thepolice or all primary users of the assistant device. Unusual activitycan include recognizing a weapon in the video input. Locations canindicate attributes requiring privacy; rules also can be assigned basedon the location associated with the devices. For example, a rule relatedto a bathroom location can prohibit the use of video devices. Thus, whena developer assigns a selectable item a functionality of turning on allcameras in the environment, a rule can automatically be generatedpreventing the video devices in the home with a bathroom location fromturning on in response to the selection of the selectable item. A ruleassociated with the “unlock lock” selectable item can include a signalmonitoring service which determines a user's proximity to the locationof the device. Signal monitoring services can determine that a user'sdevice is too far away from the location of the device and can preventthe functionality from being implemented. For example, the user may beprevented from unlocking the lock when, based on the distance determinedby the signal monitoring service, the user's device is more than athreshold distance away from the location. In an example, when theuser's device is determined using the signal monitoring service not tobe in the environment, then the rule may be set to prevent the user fromturning on appliances such as the coffee maker. In some embodiments, thethreshold distance of the user's device to the environment must be metto allow the selectable item of the application on the user's device tooperate one or more associated devices.

FIG. 5 illustrates an embodiment of an application development screen ofthe Home API. In at least one embodiment, the developer loads a graphic501, selects a button on the graphic 502, and via the Home API interfaceselects corresponding functionality of one or more devices. For example,a developer can load a graphic 501, select a transport icon 502 on thegraphic, then assign functionalities and/or rules. The area selected canthen be set to perform the functionalities and/or rules when clicked bythe user. For example, the transport icon can be configured to cause theassistant device to turn on the television devices to the traffic newschannel. In another example, the transport icon can be configured tocause the assistant device to provide a traffic report.

In at least one embodiment, the Home API allows the developer to buildthe graphic screen using the Home API GUI. The Home API can include afeature of automatic programming and can be used to generate programmingcode based on the selected rules. In an example, the graphic 501 can bebuilt using the Home API GUI.

In at least one embodiment, machine learning can be used to identifysections of a graphic intended to operate as selectable items (e.g.,buttons). The graphic represents a layout having one or more graphicalinterface elements (GUI). The graphic can be analyzed to determine thepotential selectable items by one or more of detecting edges, changes incolor between the background of the graphic and the portion of thegraphic that corresponds to the selectable item, identifying the shapes,or identifying a pattern change. In an embodiment, the section can beidentified based on a change in color between a background color of thegraphic and a color of the section of the graphic. In an example, abutton can be identified by determining that the graphic has an itemwhich is a square shape. In another example, a button can be identifiedin a graphic by determining the graphic has a color different from thecolor primarily used in the graphic. The selectable item can bedetermined by determining an interruption of pattern in the graphic. Forexample, for a graphic with a striped pattern having portions that arenot striped, the determination algorithm can identify the not stripedportions and determine those portions to be selectable items. In atleast one embodiment, determining a potentially selectable item can bedone by one or more of an interpretation tree, geometric hashingalgorithm, speech up robust features algorithm, pose clustering, objectrecognition algorithm, or another algorithm.

The Home API can also identify objects of the graphical content in thegraphic. The objects can be identified using pattern recognition,machine learning, data mining, or knowledge discovery. The graphic canbe analyzed to determine the objects depicted. For example, a graphichaving a picture of a train can be analyzed, and it can be determinedthat the graphic depicts a train. In an example where a train and a bedare depicted in the graphic, the Home API can identify the two objects(train and bed). In at least one embodiment, the objects can bedetermined to be selectable items.

In at least one embodiment, the Home API identifies the selectable itemsfirst then identifies the objects within the selectable items (e.g.,buttons). For example, a graphic which has two squares, the first squarewith a picture of a train and the second square with a picture of thebed, can be analyzed to determine that the squares are selectable items,and then it can be determined that the object train is in the firstsquare and the object bed is in the second square. In at least oneembodiment, the Home API identifies the selectable items and the objectssimultaneously, in parallel, and/or identifies the objects beforeidentifying selectable items. The objects can be determined to beassociated with a functionality and/or device by analyzing a position ofthe graphical content within the section, a size of the graphicalcontent in relation a size of the section, a relationship between theposition of the graphical content with other graphical content withinthe section, a color of the graphical content, or the color of thesection of the graphic.

The Home API can also identify the text depicted on the graphic. Forexample, the Home API can determine that a graphic has the words “Open”“Home,” or “Exit.” It can be determined that the identified wordsrepresent a selectable item. In at least one embodiment, thisdetermination can be made by referring to a database of words havingwords which are selectable items. In at least one embodiment, thecharacteristics of the text can be used to determine information aboutthe text and/or the selectable items. Characteristics of the textualcontent including one or more of a meaning of the textual content, asize of the textual content, a font of the textual content, arelationship between a position of the textual content and the positionof the graphical content, or the position of the textual content withinthe section.

In one embodiment, it can be determined whether the text is a selectableitem based on context derived. For example, the words of the text “Ibetter start planning my menu for Thanksgiving dinner” can be analyzed,and it can be determined that the text “menu” is unlikely to represent aselectable item because of the relationship to other words in thetextual content. The text of a selectable item can be analyzed todetermine the functionalities and/or related devices. The analysis canconsider the size of the text, for example larger text can be determinedto be more relevant and have a higher weight than smaller text indetermining the associated functionalities and/or devices. The positionof the text can be analyzed to determine the font, and the font can beconsidered when determining the associated functionalities and/ordevices. Certain fonts can be determined to be associated with specificfunctionalities. For example, the Times New Roman or Georgia fonts canbe determined to be fonts which give the text higher relevance. Thus, aselectable item that includes the text “click here to turn off lights”where the “turn off lights” is in a Times New Roman font and the “clickhere” is in a different font, it can be determined that the “turn offlights” is the more relevant text. The size of the font can also beconsidered in determining information about the text.

In at least one embodiment, the Home API can identify the text on agraphic. The text can be analyzed to determine potential associatedfunctionality and/or associated one or more devices. In someembodiments, the meaning or context of the text can be derived, forexample, the meaning or context can include a dictionary meaning, abrand, a device, a functionality, etc. The meaning can also include anassociated relevance level, for example, it can be determined whetherthe level of relevance the meaning has in determining the functionality,device type, or a device associated with the selectable item. In oneembodiment, the text identified within a selectable item can be used todetermine the associated functionality and/or associated one or moredevices. In some embodiments, the text identified on the graphic but notwithin the selectable items can be used to determine associatedfunctionality and/or associated one or more devices for the selectableitems. In some embodiments, the objects identified on the graphic can beused to determine the functionality and/or one or more devices for theselectable items.

A graphic with a square with the word “menu” and food objects outsidethe boundaries of the square (e.g., apple, banana, potato, chicken,etc.) can be analyzed to identify the square as a selectable item, theword “menu” can be recognized, and it can be determined that theselectable item is related to that word “menu.” The food objects can beidentified and used to provide context for the “menu” selectable item. Amachine learning algorithm can be used to determine that the “menu”selectable item is food-related. The machine learning algorithm can usethe food objects to classify the selectable item and identify theassociated functionality and/or device category. In at least oneembodiment, the Home API can allow the developer to correct wrongdeterminations. The machine learning algorithm can be set to learn fromthe developer corrections.

In at least on embodiment, the Home API has access to a database storinginformation about selectable items and the associated functionalitiesand/or device categories. For example, the database can include a recordof train-related selectable items and store the associated functionalityof providing transit information. In another example, the database canstore a record of light-related selectable items, the associatedfunctionality of “on” and the associated device category of lights.

The Home API can identify the selectable items and provide a developerwith a screen allowing the developer to configure the functionalitiesand/or devices of the selectable items; see FIGS. 3-4. In at least oneembodiment, the Home API automatically configures the selectable itemswith the determined functionalities, associated device categories,and/or device. The Home API can automatically generate the codenecessary to allow the selectable items to cause the assistant device totransmit commands to determined functionalities, associated devicecategories, and/or devices. In at least one embodiment, the Home API canaccess a database of adapters associated with the determinedfunctionalities, associated device categories, and/or devices.

In an example, a graphic can have a brand (company name) listed on thelayout, a square with the word “on” and a picture of a light bulb. Thecharacteristics of the graphic can be analyzed to determine thatincludes a square, and the square can further be identified as being aselectable item. It can further be determined that the graphic contentincludes text representing a brand. The position on the graphic inrelation to other content, relations to other content, color, font,size, and definition can be used to determine that the text is a brand.For example, the definition can be used to determine that one of themeanings of the word is a trademark, and then the color, size and fontof the text can be analyzed to determine how closely the text matchesthe trademark. The relationship to the light bulb depiction can also beidentified; for example, if the brand is on the light bulb, it can bedetermined that the brand represents the brand of the light bulb. If thelight bulb is far from the text it can be determined that likely thetext does not describe the brand of light bulb. Additionally, the sizeof the text can aid in determining the relationship; for example, if thetext of the brand is very large, then it can be determined that thebrand represents all the devices which are to be associated with theselectable items. Likewise, if the large text was “Kitchen,” it can bedetermined that all or most selectable items on the graphic are relatedto the kitchen.

The text “on” can also be used to determine the functionality associatedwith the selectable item. The text can be analyzed to determine theposition in proximity to the light bulb depiction and the position onthe selectable item. The size and the font can also be used to determinethe functionality and/or associated device. For example, if the font islarge, then it is likely that the functionality associated with theselectable item is “on.” The other nearby text can also be considered.For example, if the text in the selectable item next to the “on” text is“off,” then it can be determined that the functionality is both on andoff. Alternatively, depending on the size of the text, it can bedetermined that the “on” and “off” text has little relevance to theassociated functionality to the selectable item.

The depiction of the light bulb object can also be analyzed to determinethe functionality, device type, and/or device. The light bulb depictioncan be analyzed and determined to be associated with a lighting deviceand/or device category. The characteristics of the object can beanalyzed. The proximity of the object to the text and/or other depictedobjects, the size of the object, the size of the object in relation tothe objects, the color, or positioning to the selectable item can beused to understand the intended functionalities and/or devicesassociated with the selectable item. The object can also be determinedto be associated with an object type. Object types can includedescriptions and categories of objects, for example, food, lights,curtains, kitchen, office, television, speakers, etc. Each object typecan be associated with a device type such as lighting devices, kitchendevices, office devices, etc.

The characteristics of the object can be used to determine the relevanceof the object and can help determine the intended functionality and/ordevice. The determination algorithm can factor the relevance and/orweight of the information it collects to determine the intention. Alarge light bulb object taking up most of the space in the selectableitem can be determined to have a high likelihood of the selectablebutton being related to a light device. Alternatively, a very smalllight bulb surrounded by other objects all within the borders of theselectable device can be determined to have little relevance todetermining the functionalities and/or devices associated with theselectable item.

In the example, based on the size of the light bulb, the “on” text andthe brand name, the algorithm can determine that the functionality anddevices associated with the selectable button are turning on lightingdevices. It can be further determined that the lighting devicesassociated with the selectable item are of a specific brand. In anexample, these functionalities and devices can automatically be assignedto the selectable item of the graphic. Thus, when a user uses theapplication having the graphic, the user will need to select theselectable item, and the device will turn on the lights of the specificbrand in the user's environment.

In another example, a graphic having a square with the word “on” and apicture of a light bulb can be automatically assigned the square as aselectable item with the functionality of “on” in association with thelight category. Thus, when a user uses the application having thisgraphic, the user will select the selectable item, thereby causing thedevices in the home having the light category to turn on.

In some embodiments, the color of the depicted object can be used todetermine the functionality and/or the device. For example, a depictionof a light bulb that is black in color can be determined to represent afunctionality of “off,” while a white light bulb can be determined torepresent the functionality of “on.” In some embodiments the relevanceof the color of the depicted object can be identified by identifying thecolors of other depicted objects in the same or other selectable items.For example, if all depicted objects on the graphic are black, then theblack coloring of the light bulb can be determined to not be relevant.

In some embodiments, the graphic can have associated metadata which canbe used to identify selectable text and/or the associatedfunctionalities and/or devices. In at least one embodiment, thedeveloper can configure the application being created via the Home APIto collect information. The Home API can allow the developer to collectinformation about the devices in its proximity and to control thefunctions of only those devices. The Home API can present the developerwith information, which can be collected via a user device, which willrun the corresponding app. Information such as GPS location, deviceproximity information, and/or assistant device proximity can becollected. The developer can create rules associated with the gatheredinformation via a user device and functionalities of devices and/orassistant devices. For example, the developer can select that when thecondition of the user device being within the proximity of 10 feet fromthe Home API is met, the living room lights are turned on.

As demonstrated by FIG. 3, the Home API presents high-level commands ina natural and intuitive format. The Home API developer is not requiredto understand the nuances of the Ambient OS. The developer simply needsto select high-level commands, mechanisms, policies, contexts,environment models, and/or components (e.g., microservices, etc.). TheHome API can generate the necessary code which allows it to communicatewith the assistant device and/or the devices.

The Ambient OS can include subsystems such as a service registry, acomponent manager, an event bus, or a user manager. The service registrycan further allow the Home API to register and browse microservices. Thecomponent manager can be responsible for validating, installing,uninstalling, and browsing components, as well as providing the runtimeenvironment to instantiate these components. In at least one embodiment,a component can be a package that includes resources that can bedeployed into Ambient OS. In an embodiment, the event bus can implementcommunication channels. A user manager can be configured to providefunctionality to create, browse, edit, and remove users. The Ambient OScan be configured to be transport agnostic and rely on Gateways toimplement the underlying protocols to communicate with the system.Microservices can be decoupled from the transport so the same code workswith different protocols. Ambient OS provides the following gateways.

The Ambient OS can further include WebSocketGateways, HTTPGateway,CloudWebSocketGateway, and entity registry. The Service Registry canmaintain a list of running microservices. The WebSocketGateway: canreply on WebSockets as the communication protocol among components. TheHTTPGateway can be configured to serve WebApps that leverage Ambient OSmicroservices and expose an HTML-based GUI to users. For example, aWebApp timer can display an HTML page with a counter. This HTML page caninteract with a timer service to set the countdown time as well as startand stop the timer. Furthermore, the page can listen to a channel to geta notification from the timer service when time is up and update theuniversal interface accordingly/

In at least one embodiment, an administrative console permits theconfiguration of the Home API. The administrative console can be used tocreate user accounts and maintain the types of services that thedevelopers can access. For example, the administrative console can beused to set up permissions for developers. For instance, the superdevelopers can be allowed access to all microservices while otherdevelopers can only be configured to have access to specificmicroservices and/or tools.

FIG. 6 illustrates an example of an assistant device. In FIG. 6,assistant device 101 includes a processor 605, memory 610, touchscreendisplay 625, speaker 615, microphone 635, as well as other types ofhardware such as non-volatile memory, an interface device, camera,radios, etc. to implement communication management logic 630 providingthe techniques disclosed herein. Various common components (e.g., cachememory) are omitted for illustrative simplicity. The assistant device isintended to illustrate a hardware device on which any of the componentsdescribed in the example of FIGS. 1-5 (and any other componentsdescribed in this specification) can be implemented. The components ofthe assistant device can be coupled together via a bus or through someother known or convenient device.

The processor 605 may be, for example, a microprocessor circuit such asan Intel Pentium microprocessor or Motorola power PC microprocessor. Oneof skill in the relevant art will recognize that the terms“machine-readable (storage) medium” or “computer-readable (storage)medium” include any type of device that is accessible by the processor.Processor 605 can also be circuitry such as an application specificintegrated circuits (ASICs), complex programmable logic devices (CPLDs),field programmable gate arrays (FPGAs), structured ASICs, etc.

The memory is coupled to the processor by, for example, a bus. Thememory can include, by way of example but not limitation, random accessmemory (RAM) such as dynamic RAM (DRAM) and static RAM (SRAM). Thememory can be local, remote, or distributed.

The bus also couples the processor to the non-volatile memory and driveunit. The non-volatile memory is often a magnetic floppy or hard disk; amagnetic-optical disk; an optical disk; a read-only memory (ROM) such asa CD-ROM, EPROM, or EEPROM; a magnetic or optical card; or another formof storage for large amounts of data. Some of this data is oftenwritten, by a direct memory access process, into memory during theexecution of software in the computer. The non-volatile storage can belocal, remote or distributed. The non-volatile memory is optionalbecause systems can be created with all applicable data available inmemory. A typical computer system will usually include at least aprocessor, memory, and a device (e.g., a bus) coupling the memory to theprocessor.

The software can be stored in the non-volatile memory and/or the driveunit. Indeed, storing an entire large program in memory may not even bepossible. Nevertheless, it should be understood that for software torun, it may be necessary to move the software to a computer-readablelocation appropriate for processing, and, for illustrative purposes,that location is referred to as memory in this application. Even whensoftware is moved to memory for execution, the processor will typicallymake use of hardware registers to store values associated with thesoftware and make use of a local cache that, ideally, serves toaccelerate execution. As used herein, a software program can be storedat any known or convenient location (from non-volatile storage tohardware registers).

The bus also couples the processor to the network interface device. Theinterface can include one or more of a modem or network interface. Thoseskilled in the art will appreciate that a modem or network interface canbe considered to be part of the computer system. The interface caninclude an analog modem, an ISDN modem, a cable modem, a token ringinterface, a satellite transmission interface (e.g., “direct PC”), orother interface for coupling a computer system to other computersystems. The interface can include one or more input and/or outputdevices. The input and/or output devices can include, by way of examplebut not limitation, a keyboard, a mouse or other pointing device, diskdrives, printers, a scanner, and other input and/or output devices,including a display device. The display device can include, by way ofexample but not limitation, a cathode ray tube (CRT), a liquid crystaldisplay (LCD), or some other applicable known or convenient displaydevice.

In operation, the assistant device can be controlled by operating systemsoftware that includes a file management system, such as a diskoperating system. The file management system is typically stored in thenon-volatile memory and/or drive unit and causes the processor toexecute the various acts required by the operating system to input andoutput data, and to store data in the memory, including storing files onthe non-volatile memory and/or drive unit.

Some items of the detailed description may be presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is, here and generally,conceived to be a self-consistent sequence of operations leading to adesired result. The operations are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electronic or magnetic signals capableof being stored, transferred, combined, compared, and/or otherwisemanipulated. It has proven convenient at times, principally for reasonsof common usage, to refer to these signals as bits, values, elements,symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, as apparent from the followingdiscussion, those skilled in the art will appreciate that throughout thedescription, discussions utilizing terms such as “processing” or“computing” or “calculating” or “determining” or “displaying” or“generating” or the like refer to the action and processes of a computersystem or similar electronic computing device that manipulates andtransforms data represented as physical (electronic) quantities withinthe computer system's registers and memories into other data similarlyrepresented as physical quantities within the computer system's memoriesor registers or other such information storage, transmission, or displaydevices.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatuses to perform the methods of some embodiments. The requiredstructure for a variety of these systems will be apparent from thedescription below. In addition, the techniques are not described withreference to any particular programming language, and variousembodiments may thus be implemented using a variety of programminglanguages.

In further embodiments, the assistant device operates as a standalonedevice or may be connected (e.g., networked) to other machines. In anetworked deployment, the assistant device may operate in the capacityof a server or of a client machine in a client-server networkenvironment or may operate as a peer machine in a peer-to-peer (ordistributed) network environment.

In some embodiments, the assistant devices include a machine-readablemedium. While the machine-readable medium or machine-readable storagemedium is shown in an exemplary embodiment to be a single medium, theterm “machine-readable medium” and “machine-readable storage medium”should be taken to include a single medium or multiple media (e.g., acentralized or distributed database and/or associated caches andservers) that store the one or more sets of instructions. The term“machine-readable medium” and “machine-readable storage medium” shouldalso be taken to include any medium that is capable of storing,encoding, or carrying a set of instructions for execution by themachine, and which causes the machine to perform any one or more of themethodologies or modules of the presently disclosed technique andinnovation.

In general, the routines executed to implement the embodiments of thedisclosure may be implemented as part of an operating system or aspecific application, component, program, object, module, or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions set at various times invarious memory and storage devices in a computer that, when read andexecuted by one or more processing units or processors in a computer,cause the computer to perform operations to execute elements involvingvarious aspects of the disclosure.

Moreover, while embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that the various embodiments are capable of beingdistributed as a program product in a variety of forms, and that thedisclosure applies equally, regardless of the particular type ofmachine- or computer-readable media used to actually effect thedistribution.

Further examples of machine-readable storage media, machine-readablemedia, or computer-readable (storage) media include, but are not limitedto, recordable type media such as volatile and non-volatile memorydevices, floppy and other removable disks, hard disk drives, opticaldisks (e.g., Compact Disc Read-Only Memory (CD-ROMS), Digital VersatileDiscs (DVDs), etc.), among others, and transmission type media such asdigital and analog communication links.

In some circumstances, operation of a memory device, such as a change instate from a binary one to a binary zero or vice-versa, for example, maycomprise a transformation, such as a physical transformation. Withparticular types of memory devices, such a physical transformation maycomprise a physical transformation of an article to a different state orthing. For example, but without limitation, for some types of memorydevices, a change in state may involve an accumulation and storage ofcharge or a release of stored charge. Likewise, in other memory devices,a change of state may comprise a physical change or transformation inmagnetic orientation or a physical change or transformation in molecularstructure, such as from crystalline to amorphous or vice-versa. Theforegoing is not intended to be an exhaustive list in which a change instate for a binary one to a binary zero or vice-versa in a memory devicemay comprise a transformation, such as a physical transformation.Rather, the foregoing is intended as illustrative examples.

A storage medium may typically be non-transitory or comprise anon-transitory device. In this context, a non-transitory storage mediummay include a device that is tangible, meaning that the device has aconcrete physical form, although the device may change its physicalstate. Thus, for example, non-transitory refers to a device remainingtangible despite this change in state.

The foregoing description of various embodiments of the claimed subjectmatter has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit the claimedsubject matter to the precise forms disclosed. Many modifications andvariations will be apparent to one skilled in the art. Embodiments werechosen and described in order to best describe certain principles andpractical applications, thereby enabling others skilled in the relevantart to understand the subject matter, the various embodiments and thevarious modifications that are suited to the particular usescontemplated.

While embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that the various embodiments are capable of beingdistributed as a program product in a variety of forms and that thedisclosure applies equally regardless of the particular type of machine-or computer-readable media used to actually effect the distribution.

FIGS. 7a-7l illustrate embodiments of the assistant device. Although theabove Detailed Description describes certain embodiments and the bestmode contemplated, no matter how detailed the above appears in text, theembodiments can be practiced in many ways. Details of the systems andmethods may vary considerably in their implementation details whilestill being encompassed by the specification. As noted above, particularterminology used when describing certain features or aspects of variousembodiments should not be taken to imply that the terminology is beingredefined herein to be restricted to any specific characteristics,features, or aspects of the disclosed technique with which thatterminology is associated. In general, the terms used in the followingclaims should not be construed to limit the disclosure to the specificembodiments disclosed in the specification, unless those terms areexplicitly defined herein. Accordingly, the actual scope of thetechnique encompasses not only the disclosed embodiments but also allequivalent ways of practicing or implementing the embodiments under theclaims.

The language used in the specification has been principally selected forreadability and instructional purposes, and it may not have beenselected to delineate or circumscribe the inventive subject matter. Itis therefore intended that the scope of the technique be limited not bythis Detailed Description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of variousembodiments is intended to be illustrative, but not limiting, of thescope of the embodiments, which is set forth in the following claims.

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thescope of the invention. Accordingly, the invention is not limited exceptas by the appended claims.

We claim:
 1. A method for seamlessly and automatically building anapplication using an application programming interface (API) tointerface with one or more devices within an environment with anassistant device having artificial intelligence (AI), comprising:receiving, via a processor, a graphic representing a layout of one ormore graphical user interface (GUI) elements for the application toinstruct the assistant device to control the one or more devices withinthe environment; identifying, by the processor, a section of the graphiccorresponding to the one or more GUI elements for the application thatis to be a selectable item on the application to implement afunctionality to be performed within the environment using the assistantdevice, wherein the section of the graphic is identified based on achange in color between a background color of the graphic and a sectioncolor of the graphic; identifying, by the processor, a graphical contentwithin the section of the graphic; determining, by the processor, objectcharacteristics of the graphical content within the section of thegraphic corresponding to the one or more GUI elements for theapplication that is to be the selectable item on the application toimplement the functionality to be performed within the environment usingthe assistant device, wherein the object characteristics of thegraphical content within the section of the graphic includes a type ofobject corresponding to the graphical content within the section of thegraphic, a position of the graphical content within the section of thegraphic, a size of the graphical content within the section of thegraphic in relation to a size of the section of the graphic, arelationship between a position of the graphical content within thesection of the graphic with other graphical content within the sectionof the graphic, or a color of the graphical content within the sectionof the graphic; identifying, by the processor, a textual content withinthe section of the graphic; determining, by the processor, textcharacteristics of the textual content within the section of the graphiccorresponding to the one or more GUI elements for the application thatis to be the selectable item on the application to implement thefunctionality to be performed within the environment using the assistantdevice, wherein the text characteristics of the textual content withinthe section of the graphic includes one or more of a meaning of thetextual content within the section of the graphic, a size of the textualcontent within the section of the graphic, a font of the textual contentwithin the section of the graphic, a relationship between a position ofthe textual content within the section of the graphic and the positionof the graphical content within the section of the graphic, or theposition of the textual content within the section of the graphic;determining, by the processor using a database, the functionality to beperformed within the environment using the assistant device based on thetext characteristics of the textual content within the section of thegraphic and the object characteristics of the graphical content withinthe section of the graphic; identifying, by the processor, the one ormore devices capable of performing the functionality within theenvironment; and generating, by the processor, the application havingthe section of the graphic, wherein the section of the graphic, whenselected, instructs the assistant device to cause the identified one ormore devices to perform the functionality within the environment.
 2. Amethod, comprising: receiving, via a processor, a graphic representing alayout of one or more graphical user interface (GUI) elements for anapplication to instruct an assistant device to control one or moredevices within an environment; identifying, by the processor, a sectionof the graphic corresponding to the one or more GUI elements for theapplication that is to be a selectable item on the application toimplement a functionality to be performed within the environment usingthe assistant device; identifying, by the processor, a graphical contentwithin the section of the graphic; determining, by the processor,characteristics of the graphical content within the section of thegraphic, wherein the graphical content within the section of the graphicincludes a depiction of an object or a depiction of a text; determining,by the processor using a database, the functionality to be performedwithin the environment using the assistant device based on thecharacteristics of the graphical content within the section of thegraphic and device categories; identifying, by the processor, the one ormore devices capable of performing the functionality within theenvironment; and generating, by the processor, the application havingthe section of the graphic, wherein the section of the graphic, whenselected, instructs the assistant device to cause the identified one ormore devices to perform the functionality within the environment.
 3. Themethod of claim 2, wherein the characteristics of the graphical contentwithin the section of the graphic include a position of the graphicalcontent within the section of the graphic.
 4. The method of claim 3,wherein the characteristics of the graphical content within the sectionof the graphic include a size of the graphical content within thesection of the graphic in relation to a size of the section of thegraphic.
 5. The method of claim 2, wherein the characteristics of thegraphical content within the section of the graphic include meaning ofthe depiction of the text.
 6. The method of claim 2, wherein thecharacteristics of the graphical content within the section of thegraphic include a color of the depiction of the object found on agraphic outside the section of the graphic.
 7. The method of claim 2,wherein the characteristics of the graphical content within the sectionof the graphic include a color of the section of the graphic.
 8. Themethod of claim 2, wherein the characteristics of the graphical contentwithin the section of the graphic include a color of the depiction ofthe object found in the section of the graphic.
 9. The method of claim2, wherein determining the functionality to be performed within theenvironment using the assistant device includes a first supervisedlearning algorithm.
 10. The method of claim 9, wherein the graphicalcontent within the section of the graphic includes the depiction of theobject and the depiction of the text.
 11. The method of claim 9, whereinidentifying the one or more devices capable of performing thefunctionality within the environment includes a second supervisedlearning algorithm.
 12. An electronic device, comprising: one or moreprocessors; a database storing characteristics of a graphical content, afunctionality, and device categories; and a memory storing instructions,execution of which by the one or more processors cause the electronicdevice to: receive a graphic representing a layout of one or moregraphical user interface (GUI) elements for an application to instructan assistant device to control one or more devices within anenvironment; identify a section of the graphic corresponding to the oneor more GUI elements for the application that is to be a selectable itemon the application to implement the functionality to be performed withinthe environment using the assistant device; identify the graphicalcontent within the section of the graphic; determine characteristics ofthe graphical content within the section of the graphic, wherein thegraphical content within the section of the graphic includes a depictionof an object or a depiction of a text; determine, using the database,the functionality to be performed within the environment using theassistant device based on the characteristics of the graphical contentwithin the section of the graphic and the device categories; identifythe one or more devices capable of performing the functionality withinthe environment; and generate the application having the section of thegraphic, wherein the section of the graphic, when selected, instructsthe assistant device to cause the identified one or more devices toperform the functionality within the environment.
 13. The electronicdevice of claim 12, wherein the characteristics of the graphical contentwithin the section of the graphic include a position of the graphicalcontent within the section of the graphic.
 14. The electronic device ofclaim 13, wherein the characteristics of the graphical content withinthe section of the graphic include a size of the graphical contentwithin the section of the graphic in relation to a size of the sectionof the graphic.
 15. The electronic device of claim 12, wherein thecharacteristics of the graphical content within the section of thegraphic include meaning of the depiction of the text.
 16. The electronicdevice of claim 12, wherein the characteristics of the graphical contentwithin the section of the graphic include a color of the depiction ofthe object found on a graphic outside the section of the graphic. 17.The electronic device of claim 12, wherein the characteristics of thegraphical content within the section of the graphic include a color ofthe section of the graphic.
 18. The electronic device of claim 12,wherein the characteristics of the graphical content within the sectionof the graphic include a color of the depiction of the object found inthe section of the graphic.
 19. The electronic device of claim 12,wherein the determination of the functionality to be performed withinthe environment using the assistant device includes a first supervisedlearning algorithm.
 20. The electronic device of claim 19, wherein thegraphical content within the section of the graphic includes thedepiction of the object and the depiction of the text.
 21. Theelectronic device of claim 19, wherein the identification of the one ormore devices capable of performing the functionality within theenvironment includes a second supervised learning algorithm.